Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[VL] Rebase upstream velox to 11/7 #3624

Merged
merged 1 commit into from
Nov 7, 2023
Merged

[VL] Rebase upstream velox to 11/7 #3624

merged 1 commit into from
Nov 7, 2023

Conversation

JkSelf
Copy link
Contributor

@JkSelf JkSelf commented Nov 6, 2023 •
edited
Loading

What changes were proposed in this pull request?

  1. upstream velox remove velox::core::QueryConfig::kAggregationSpillPartitionBits conf.
  2. upstream velox upgrade arrow version to 14.0.

How was this patch tested?

Pass CI

@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 6, 2023

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@JkSelf
Copy link
Contributor Author

JkSelf commented Nov 6, 2023

/Benchmark Velox

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3624_time.csv log/native_master_11_05_2023_16e287450_time.csv difference percentage
q1 32.92 34.85 1.934 105.88%
q2 24.80 25.12 0.320 101.29%
q3 38.33 40.11 1.779 104.64%
q4 37.36 37.09 -0.271 99.27%
q5 69.63 71.44 1.807 102.60%
q6 6.47 9.10 2.634 140.74%
q7 83.98 87.61 3.632 104.32%
q8 86.53 85.22 -1.314 98.48%
q9 121.95 121.12 -0.826 99.32%
q10 52.63 51.83 -0.799 98.48%
q11 19.98 19.68 -0.302 98.49%
q12 25.04 28.01 2.978 111.89%
q13 48.73 48.61 -0.117 99.76%
q14 17.23 19.14 1.914 111.11%
q15 30.52 33.03 2.501 108.19%
q16 16.31 16.10 -0.203 98.75%
q17 103.01 102.08 -0.927 99.10%
q18 148.33 149.04 0.710 100.48%
q19 14.99 18.25 3.259 121.74%
q20 31.22 30.91 -0.302 99.03%
q21 223.85 222.95 -0.906 99.60%
q22 13.29 13.47 0.174 101.31%
total 1247.09 1264.76 17.674 101.42%

@JkSelf JkSelf force-pushed the test-velox-upstream branch 3 times, most recently from be53633 to 9175184 Compare November 7, 2023 01:45
@JkSelf
Copy link
Contributor Author

JkSelf commented Nov 7, 2023

/Benchmark Velox

@JkSelf JkSelf force-pushed the test-velox-upstream branch 3 times, most recently from 04dd0fe to afee046 Compare November 7, 2023 05:22
@JkSelf
Copy link
Contributor Author

JkSelf commented Nov 7, 2023

/Benchmark Velox

@JkSelf JkSelf force-pushed the test-velox-upstream branch from b9ec388 to a72919c Compare November 7, 2023 12:09
@JkSelf JkSelf changed the title [VL] [DNM] Test upstream velox rebase [VL] Rebase upstream velox to 11/7 Nov 7, 2023
zhouyuan
zhouyuan approved these changes Nov 7, 2023
View reviewed changes
Copy link
Contributor

@zhouyuan zhouyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3624_time.csv log/native_master_11_06_2023_d57fdf26f_time.csv difference percentage
q1 33.96 33.34 -0.616 98.19%
q2 24.92 24.84 -0.075 99.70%
q3 37.83 38.31 0.472 101.25%
q4 37.21 37.40 0.191 100.51%
q5 69.51 70.83 1.317 101.89%
q6 6.44 7.74 1.304 120.25%
q7 84.30 85.97 1.671 101.98%
q8 87.31 84.68 -2.631 96.99%
q9 120.48 124.39 3.916 103.25%
q10 55.80 54.67 -1.132 97.97%
q11 20.13 19.96 -0.172 99.15%
q12 25.81 24.52 -1.292 95.00%
q13 49.46 50.37 0.909 101.84%
q14 18.14 18.50 0.366 102.02%
q15 31.63 32.09 0.461 101.46%
q16 16.20 16.51 0.312 101.92%
q17 102.98 100.90 -2.082 97.98%
q18 148.26 148.27 0.013 100.01%
q19 15.49 14.74 -0.753 95.14%
q20 30.03 30.95 0.916 103.05%
q21 222.75 225.06 2.316 101.04%
q22 13.40 13.61 0.210 101.57%
total 1252.03 1257.65 5.622 100.45%

@JkSelf JkSelf merged commit e3eff1d into apache:main Nov 7, 2023
17 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3624_time.csv log/native_master_11_06_2023_d57fdf26f_time.csv difference percentage
q1 33.40 33.34 -0.058 99.83%
q2 25.08 24.84 -0.239 99.05%
q3 37.85 38.31 0.456 101.21%
q4 35.71 37.40 1.686 104.72%
q5 69.88 70.83 0.943 101.35%
q6 8.09 7.74 -0.351 95.67%
q7 83.65 85.97 2.325 102.78%
q8 86.01 84.68 -1.324 98.46%
q9 122.42 124.39 1.977 101.61%
q10 53.58 54.67 1.086 102.03%
q11 19.65 19.96 0.307 101.56%
q12 26.88 24.52 -2.366 91.20%
q13 48.37 50.37 2.001 104.14%
q14 16.90 18.50 1.601 109.47%
q15 31.58 32.09 0.516 101.64%
q16 16.18 16.51 0.325 102.01%
q17 101.71 100.90 -0.812 99.20%
q18 147.76 148.27 0.514 100.35%
q19 14.81 14.74 -0.068 99.54%
q20 30.21 30.95 0.744 102.46%
q21 228.71 225.06 -3.645 98.41%
q22 13.25 13.61 0.353 102.66%
total 1251.68 1257.65 5.974 100.48%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_11_07_2023_time.csv log/native_master_11_06_2023_d57fdf26f_time.csv difference percentage
q1 34.38 33.34 -1.034 96.99%
q2 25.03 24.84 -0.184 99.26%
q3 38.14 38.31 0.162 100.42%
q4 37.57 37.40 -0.171 99.55%
q5 71.50 70.83 -0.677 99.05%
q6 6.26 7.74 1.482 123.68%
q7 82.22 85.97 3.750 104.56%
q8 86.95 84.68 -2.265 97.39%
q9 119.81 124.39 4.584 103.83%
q10 51.26 54.67 3.404 106.64%
q11 19.73 19.96 0.227 101.15%
q12 24.39 24.52 0.128 100.53%
q13 50.30 50.37 0.072 100.14%
q14 17.67 18.50 0.836 104.73%
q15 30.35 32.09 1.740 105.73%
q16 16.20 16.51 0.307 101.89%
q17 101.51 100.90 -0.610 99.40%
q18 148.26 148.27 0.007 100.00%
q19 16.17 14.74 -1.430 91.16%
q20 30.31 30.95 0.638 102.10%
q21 224.88 225.06 0.180 100.08%
q22 14.08 13.61 -0.479 96.60%
total 1246.98 1257.65 10.669 100.86%

@EpsilonPrime
Copy link
Contributor

Should the pom.xml's arrow.version be updated from 12 to 14 as well?

@JkSelf
Copy link
Contributor Author

JkSelf commented Nov 8, 2023

Should the pom.xml's arrow.version be updated from 12 to 14 as well?

It seems that the dependency exists between the Arrow versions in native and Java. On the Java side, there is a requirement for ArrowAbiUtil, which can be fulfilled by using Arrow version 12. There is no need to upgrade to Arrow version 14. cc @zhztheplayer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhouyuan zhouyuan zhouyuan approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@JkSelf @GlutenPerfBot @EpsilonPrime @zhouyuan